Image processing apparatus, image processing method, and program

ABSTRACT

An image processing apparatus which performs processing on an input moving image including a plurality of access units arranged every first period, includes a motion vector calculation unit which calculates a motion vector of an object included in the input moving image every second period, a motion vector conversion unit which, converts the motion vector by multiplying the calculated motion vector by a predetermined gain, a frame compensation unit which generates an output moving image including a plurality of access units arranged every third period by performing frame compensation processing of converting or compensating for the access units on the input moving image so that the object moves in accordance with the motion vector which has been subjected to the conversion, and a gain calculation unit which calculates the gain in accordance with brightness in a user environment and supplies the obtained gain to the motion vector conversion unit.

CROSS REFERENCES TO RELATED APPLICATIONS

The present invention contains subject matter related to Japanese PatentApplication JP 2007-260840 filed in the Japanese Patent Office on Oct.4, 2001, the entire contents of which are incorporated herein byreference,

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing apparatuses, imageprocessing methods, and programs, and more particularly relates to animage processing apparatus capable of appropriately displaying a moviesource irrespective of change of brightness of an environment, an imageprocessing method, and a program.

2. Description of the Related Art

In general, when a movie is shown at a movie theater, identical twoframes (fields) are displayed in 48 Hz. Therefore, a frame arrangementperiod of image data included in a movie source corresponds to 24 Hz.That is, the image data included in the movie source has 24 frames.

On the other hand, a display period of general television broadcastingreceivers corresponds to 60 Hz (120 Hz). Therefore, when image data of amovie source is displayed using a. general television receiver, theimage data having 24 frames is necessary to be converted into image datahaving 60 frames (120 frames).

Japanese Unexamined Patent Application Publication No. 2006-066986discloses a 2-3 pull down technique as an example of a technique ofconverting image data having 24 frames into image data having 60 frames(120 frames). Furthermore, a technique of compensating for frames bycalculating a motion vector of an object so that the object moves inaccordance with the motion vector has been proposed (hereinafterreferred to as a “motion-vector utilizing processing”).

SUMMARY OF THE INVENTION

However, there arises a problem in that when a user views a movie sourcewhich has been subjected to 2-3 pull down processing using a televisionbroadcasting receiver, for example, in a room environment in which aroom is brighter than general movie theaters, a highly-visible judder isgenerated. On the other hand, when a user views a movie source which hasbeen subjected to the motion vector utilizing processing using atelevision broadcasting receiver, for example, in a room environment inwhich a room is as dark as general movie theaters, there arises aproblem in that the movie source is shown like a video image rather thanbeing a movie, that is, realistic sensation which is one ofcharacteristics of movies is deteriorated.

Accordingly, it is desirable to appropriately display a movie scarceirrespective of change of brightness of an environment.

According to an embodiment of the present invention, there is providedan image processing apparatus which performs processing on an inputmoving image including a plurality of access units arranged every firstperiod. The image processing apparatus includes motion vectorcalculation means for calculating a motion vector of an object includedin the input moving image every second period, motion vector conversionmeans for converting the motion vector by multiplying the motion vectorcalculated using the motion vector calculation means by a predeterminedgain, frame compensation means for generating an output moving imageincluding a plurality of access units arranged every third period byperforming frame compensation processing of converting or compensatingfor the access units on the input moving image so that the object movesin accordance with the motion vector which has been subjected to theconversion performed using the motion vector conversion means, and gaincalculation means for calculating the gain in accordance with brightnessin a user environment where a user views the output moving image andsupplying the obtained gain to the motion vector conversion means.

The image processing apparatus may further include brightness detectionmeans for detecting the brightness in the user environment. The gaincalculation means calculates the gain in accordance with the brightnessdetected using the brightness detection means.

The image processing apparatus may further includes correction means forcorrecting the gain calculated using the gain calculation means inaccordance with an instruction issued by the user.

An image processing method according to another embodiment of thepresent invention and a program according to a further embodiment of thepresent invention are associated with the image processing apparatusaccording to the embodiment of the present invention.

The image processing method according to the other embodiment of thepresent invention and the program according to the further embodiment ofthe present invention perform the following processing on an inputmoving image including a plurality of access units arranged every firstperiod. A motion vector of an object included in the input moving imageis calculated every second period, the motion vector is converted bymultiplying the calculated motion vector by a predetermined gain, anoutput moving image including a plurality of access units arranged everythird period, is generated by performing frame compensation processingof converting or compensating for the access units on the input movingimage so that the object moves in accordance with the motion vectorwhich has been subjected to the conversion. Here, the gain is calculatedin accordance with brightness in a user environment where a user viewsthe output moving image.

Accordingly, image processing performed on a moving image of a moviesource is realised. In particular, the movie source is appropriatelydisplayed irrespective of change of brightness of an environment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of a functionalconfiguration of an image processing apparatus according to anembodiment of the present invention;

FIG. 2 is a timing chart illustrating an example of an input signal;

FIG. 3 is a timing chart illustrating a movement of an object in a filmas an input signal;

FIG. 4 is a timing chart illustrating a movement of the object relativeto an output signal obtained after the image processing apparatus shownin FIG. 1 performs frame compensation processing on the input signalshown in FIG. 3 in a case where all motion vectors obtained afterconversion processing become 0 when a room turns dark, for example;

FIG. 5 is a timing chart illustrating a movement of the object relativeto an output signal obtained after the image processing apparatus shownin FIG. 1 performs the frame compensation processing on the input signalshown in FIG. 3 in a case where ail the motion vectors obtained afterthe conversion processing become equal to motion vectors which have notyet been subjected to the conversion processing when a room is lit up,for example;

FIG. 6 is a diagram illustrating an example of a function stored in again calculation unit of FIG. 1;

FIG. 7 is a timing chart illustrating a movement of the object relativeto an output signal obtained after the image processing apparatus shownin FIG. 1 performs the frame compensation processing on the input signalshown in FIG. 3 in a case where all the motion vectors corresponds toamounts of movements obtained after the conversion processing becomehalf the motion vectors which have not yet been subjected to theconversion processing when a room environment is changed so as to have abrightness in a middle of a range between brightness in the case of FIG.4 and brightness in the case of FIG. 5, for example;

FIG. 8 is a timing chart illustrating a movement of the object in thefilm represented by an input signal which has been subjected to the 2-3pull down processing;

FIG. 9 is a timing chart illustrating a movement of the object relativeto an output signal obtained after the image processing apparatus shownin FIG. 1 performs frame compensation processing on the input signalshown in FIG. 8 in a case where all motion vectors obtained afterconversion processing become 0 when a room turns dark, for example;

FIG. 10 is a timing chart illustrating a movement of the object relativeto an output signal obtained after the image processing apparatus shownin FIG. 1 performs the frame compensation processing on the input signalshown in FIG. 8 in a case where all the motion vectors obtained afterthe conversion processing become half the motion vectors which have notyet been subjected to the conversion processing when a room environmentis changed so as to nave a brightness in a middle of a range between thebrightness in the case of FIG. 4 and the brightness in the case of FIG.5, for example;

FIG. 11 is a timing chart illustrating a movement of the object relativeto an output signal obtained after the image processing apparatus shownin FIG. 1 performs the frame compensation processing on the input signalshown in FIG. 8 in a case where all the motion vectors obtained afterthe conversion processing become equal to motion vectors which have notyet been subjected to the conversion processing when a room is lit up,for example; and

FIG. 12 is a block diagram illustrating an example of an entire or apart of a hardware configuration of the image processing apparatusaccording to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before describing an embodiment of the present invention, thecorrespondence between the features of the claims and the specificelements disclosed in an embodiment of the present invention isdiscussed below. This description is intended to assure that anembodiment supporting the claimed invention is described in thisspecification. Thus, even if an element in the following embodiment isnot described as relating to a certain feature of the present invention,that does not necessarily mean that the element does not relate to thatfeature of the claims. Conversely, even if an element is describedherein as relating to a certain feature of the claims, that does notnecessarily mean that the element does not relate to other features ofthe claims.

Furthermore, this description should not be construed as restrictingthat all the embodiments of the invention are described in the claims.That is, the description does not deny the existence of embodiments ofthe present invention that are not claimed in the invention of thisapplication, i.e., the existence of embodiments of the present inventionthat in future may be claimed by a divisional application, or that maybe additionally claimed through amendments.

An image processing apparatus (an image processing apparatus shown inFIG. 1, for example) according to an embodiment of the present inventionwhich performs processing on an input moving image (a film shown in adrawing on an upper side of FIG. 2 and a film which has been subjectedto 2-3 pull down processing shown in a drawing on a lower side of FIG.2, for example) including a plurality of access units arranged everyfirst period. The image processing apparatus includes a motion vectorcalculation unit (a motion vector calculation unit 11 shown in FIG. 1,for example) configured to calculate a motion vector of an objectincluded in the input moving image every second period, a motion vectorconversion unit (a motion vector conversion unit 12 shown in FIG. 1, forexample) configured to convert the motion vector by multiplying themotion vector calculated using the motion vector calculation unit by apredetermined, gain, a frame compensation unit (a frame compensationunit 13 shown in FIG. 1, for example) configured to generate an outputmoving image including a plurality of access units arranged every thirdperiod by performing frame compensation processing of converting orcompensating for the access units on the input moving image so that theobject moves in accordance with the motion vector which has beensubjected, to the conversion performed using the motion vectorconversion unit, and a gain calculation unit (a gain calculation unit 14shown in FIG. 1, for example) configured to calculate the gain inaccordance with brightness in a user environment where a user views theoutput moving image and supplying the obtained gain to the motion vectorconversion unit.

The image processing apparatus further includes a brightness detectionunit (a brightness detector 15 shown in FIG. 1, for example) configuredto detect the brightness in the user environment. The gain calculationunit calculates the gain in accordance with the brightness detectedusing the brightness detection unit.

The image processing apparatus further includes a correction unit (auser correction unit 16 shown in FIG. 1, for example) configured tocorrect the gain calculated using the gain calculation unit inaccordance with an instruction issued by the user.

An image processing method according to another embodiment of thepresent invention and a program according to a further embodiment of thepresent invention are associated with the image processing apparatusaccording to the embodiment of the present invention. Although theprogram will be described hereinafter in detail, the program is recordedin a removable medium 111 shown in FIG. 12 or a recording medium such asa hard disk included in a storage unit 108, and is executed by acomputer having a configuration shown in FIG. 12.

According to a still further embodiment of the present invention, thereis provided a recording medium which stores the program according to oneof the embodiments of the present invention.

The image processing apparatus according to one of the embodiments ofthe present invention is usable as an entire television system or acomponent of the television system. The television system means a systemincluding at least one AV (Audio and Visual) device including atelevision broadcasting receiver.

An embodiment of the present invention will now be described withreference to the accompanying drawings.

Note that the description described below has the followingprerequisite.

Specifically, in various embodiments, which will be describedhereinafter, various image processing operations are performed on movingimages in an access unit. The “access unit” means a unit of a movingimage such as a frame and a field, and specifically, means an entire (aframe, for example) or a part (a field, for example) of each of stillimages constituting the moving image. Note that the various imageprocessing operations are performed on the moving image in a unit of aframe hereinafter for simplicity.

Furthermore, the moving image (video image) to be subjected to imageprocessing is referred to as a signal or data hereinafter.

Moreover, signals supplied to functional blocks included in an imageprocessing apparatus, which will be described with reference to FIG. 1hereinafter, are equally referred, to as input signals. That is, thesignals supplied to the functional blocks are equally referred to asinput signals irrespective of a unit of input such as a unit of a movingimage, a unit of a frame included in the moving image, and a unit of apixel (pixel value) constituting the frames. Similarly, signals outputfrom the functional blocks are equally referred, to as output signalsirrespective of a unit of output. When the unit of input or the unit ofoutput is necessary to be specified, however, the description is madeusing a specified unit (mainly, a unit of a frame), and otherwise, thedescription is made using an input signal or an output signal.

FIG. 1 is a block diagram illustrating an example of a functionalconfiguration of an image processing apparatus according to anembodiment of the present invention.

The image processing apparatus shown in FIG. 1 includes a motion vectorcalculation unit 11, a motion vector conversion unit 12, a framecompensation unit 13, a gain calculation unit 14, a brightness detector15, and a user correction unit 16.

In the image processing apparatus shown in FIG. 1, an input signal T0representing a moving image having a frame period of 24 Hz, for example,is supplied to each of the motion vector calculation unit 11 and theframe compensation unit 13.

Note that a signal having a frame period of L Hz (L is an arbitraryinteger value) is simply referred to as an “L-Hz signal” hereinafter.

The motion vector calculation unit 11 calculates a motion vector MV ofthe supplied input signal T0 every predetermined period, for example, aperiod of 24 Hz, and supplies the motion vector MV to the motion vectorconversion unit 12.

Here, motion vectors MV are calculated for individual pixel valuesconstituting a frame. Note that a certain object (represented byrectangular regions shown in FIG. 3, for example, which will bedescribed hereinafter) included in the moving image represented by theinput signal T0 is taken as an example in the description below forsimplicity, and the motion vectors MV represent vectors indicatingamounts of movement and directions of the movement of the certainobject.

In other words, a processing operation performed on pixels included inthe certain object is focused on hereinafter among processing operationsperformed on a frame. However, it is apparent that if a zero vector isincluded in the motion vectors, other pixels included in the frame aresubjected to processing operations similar to the processing operationdescribed below.

The motion vector conversion unit 12 converts a motion vector MV into amotion vector G×MV by multiplying the motion vector MV by a gain Gcalculated using the gain calculation unit 14. The obtained motionvector G×MV is supplied to the frame compensation unit 13.

The frame compensation unit 13 performs frame compensation processing onthe input signal T0 and outputs a signal T1 as an output signal obtainedas a result of the frame compensation processing. In the framecompensation processing, new frames are inserted between the framesincluded, in the input signal T0 so that the input signal T0 having anumber of frames (frame period) is converted into a signal having anincreased number of frames. By this, the 24-Hz input signal T0 isconverted into the 120-Hz output signal T1 in this embodiment.

Specifically, in this embodiment, the frame compensation unit 13performs the following frame compensation processing, for example. Asdescribed above, the motion vector MV obtained through the calculationperformed using the motion vector calculation unit 11 represents anamount of movement and a direction of the movement of the certain objectincluded in the moving image for simplicity. Therefore, the framecompensation unit 13 performs the frame compensation processing ofconverting or compensating for the frames of the input signal T0 so thatthe certain object moves in accordance with the motion vector G×MVobtained through the conversion performed using the motion vectorconversion unit 12, In this way, the 120-Hz output signal T1, that is,the output signal T1 including a plurality of frames arranged with aframe period of 120 Hz is generated.

The frame compensation processing of this embodiment will be describedin detail hereinafter with reference to FIGS. 2 to 11.

The gain calculation unit 14 calculates the gain G in accordance with abrightness P1 detected using the brightness detector 15. Note that therelationship between the brightness PI and the gain G will be describedhereinafter with reference to FIG. 6.

The brightness detector 15 detects the brightness P1 in a userenvironment in which the user views a video image corresponding to thesignal T1 output from the frame compensation unit 13, and supplies thebrightness P1 to the gain calculation unit 14.

The user correction unit 16 supplies a gain correction amount ΔGobtained in accordance with an instruction operation of the user to thegain calculation unit 14. Specifically, the user freely compensates forthe gain G using the image processing apparatus shown in FIG. 1.

Referring now to FIGS. 2 to 11, operation of the image processingapparatus shown in FIG. 1 will be described.

FIG. 2 shows a timing chart illustrating an example of the input signalT0. In FIG. 2, rectangular regions denoted by reference characters AK (Kis an arbitrary integer value) and BR in the middle portions of therectangular regions represent individual frames, and an axis of abscissadenotes time (time [sec]). Note that the frames described above will bereferred, to as frames AK and BK.

Specifically, a drawing on an upper side of FIG. 2 shows a timing chartin a case where image data having a frame period, of 24 Hz, that is,image data (hereinafter referred to as a “film”) representing a moviesource including 24 frames is supplied as the input signal T0.

A drawing on a lower side of FIG. 2 shows a timing chart in a case whereimage data (hereinafter referred to as a film which has been subjectedto the 2-3 pull down processing) obtained by converting the film by the2-3 pull down processing is supplied as the input signal T0. Here, inthe 2-3 pull down technique, the image data having a frame period of 24Hz and having frames AK and BK alternately arranged is converted intoimage data having a frame period of 60 Hz by inserting new frames AKafter the frames AK so that each of the frames AK is followed by one ofthe new frames AK which is identical to the corresponding one of theframes AK, and inserting new frames BK after the frames BK so that eachof the frames BK is followed by two of the new frames BK.

That is, the film or the film which has been subjected to the 2-3 pulldown processing is supplied as the input signal T0 in this embodiment.

An operation of the image processing apparatus shown in FIG. 1 in a casewhere the film is supplied as the input signal T0 will first bedescribed. Thereafter, another operation of the image processingapparatus shown in FIG. 1 in a case where the film which has beensubjected to the 2-3 pull down processing will be described.

As shown in FIG. 3, it is assumed that an object which is represented,by rectangular regions in FIG. 3 and which is included in the film movesto the right by α [pix] every 5/120[sec]. Specifically, in FIG. 3,rectangular regions A1, B1, and A2 represent an identical objectincluded in frames A1, B1, and A2, respectively, which uniformly movesto the right with a speed of α [pix/(5/120)sec].

In other words, FIG. 3 is a timing chart illustrating a movement of theobject. In FIG. 3, an axis of ordinate denotes time and an axis ofabscissa denotes positions of the object included in the frames in alateral direction. Reference characters MVP (P denotes an integer valueor a dash) assigned to arrow marks represent motion vectors. However,the arrow marks do not represent actual motion vectors. Note thatassuming that an angle defined by the arrow marks and a line extendingdownward in FIG. 3 is denoted by θ, sinθ relative to the arrow marksrepresents the actual motion vectors. Note that these definitionsemployed, in FIG. 3 are similarly employed in FIGS. 4 and 5 and FIGS. 7to 11, which will be described, hereinafter.

As shown in FIGS. 2 and 3, the film represented by the input signal T0corresponds to image data having a frame period of 24 Hz. In addition, aunit of processing of the motion vector calculation unit 11 is a frameperiod of 24 Hz in this embodiment as described above. Therefore, themotion vector calculation unit 11 calculates motion vectors MV forindividual frames which constitute the film. That is, in the exampleshown in FIG. 3, motion vectors MV1, MV2, and MV3 for the frames A1, B1,and A2, respectively, of the object are obtained. Here, each of themotion vectors MV1, MV2, and MV3 has a movement direction to the rightand an amount of a movement of α[pix/(5/120)sec].

In order to generate a 120-Hz output signal T1 by performing the framecompensation processing on the 24-Hz input signal T0, four frames whichare arranged with even time intervals should be inserted between each ofa pair of the frames A1 and B1 and a pair of the frames B1 and A2.

As shown in FIG. 4, for example, four frames A1′, four frames B1′, andfour frames A2′ may be inserted after the frames A1, B1, and A2,respectively, by performing processing similar to the 2-3 pull downprocessing so that the 120-Hz output signal T1 is generated.

In this case, when the frame A1 and the four frames A1′ are displayed,an edge portion of the object (represented by the rectangular regionsshown in FIG. 4) do not move from a position represented by α. Aposition of the edge portion of the object which corresponds to M (M isan arbitrary integer value) is simply referred to as an “H position”hereinafter.

When the frame B1 is displayed, the edge portion of the objectdrastically moves to a 2×α position, and while the four frames B1′ aredisplayed, the edge portion of the object does not move from the 2Δαposition. Furthermore, when the frame A2 is displayed, the edge portionof the object drastically moves to a 3×α position, and while the fourframes A2′ are displayed, the edge portion of the object does not movefrom the 3×α position.

As described above, the object does not move during a period of 5/120(=1/24) [sec], and thereafter, when a next period of 5/120 (=1/24) [sec]has passed, the object drastically moves to the right by α. That is, ahighly-visible judder is generated.

In particular, when brightness in the user environment (for example,brightness in a room, or brightness of a screen) is higher thanpredetermined brightness, that is, when the user environment is brighterthan general movie theaters, a highly-visible judder is generated. Thisis because human eyes have such a characteristic that although the humaneyes are capable of following a moving object, accuracy of thiscapability changes in accordance with brightness. Specifically, thehuman eyes have such a characteristic that as the user environment isbrighter, the human eyes follow the object moving at higher speed. Inother words, this capability corresponds to eye sensibility, and thehuman eyes have such a characteristic that assuming that as the eyesensibility is higher, the human eyes follow an object moving at higherspeed, the eye sensibility and the brightness has a constantrelationship. Such a characteristic is referred to as an “eye-followingcharacteristic” hereinafter.

For example, first to fourth frames A1′, first to fourth frames B1′, andfirst to fourth frames A2′ may be inserted after the frames A1, B1, andA2, respectively, so that the object (represented by rectangular regionsin FIG. 5) moves to the right in accordance with the motion vectors MV1,MV2, and MV3 as shown in FIG. 5, whereby the 120-Hz output signal T1 isgenerated.

In this case, when a time point in which the frame A1 is displayed isset to “0” and a position α [pix] of the object is set as a referenceposition, a video image corresponding to the output signal T1 isdisplayed such that the object moves as follows.

In the first frame A1′ which is inserted immediately after the frame A1and which is displayed at a time point of 1/120 [sec], the object movesby MV1 (=α) [pix/(5/120 ) sec]×1/120 [sec]. In the second frame A1′which is inserted immediately after the first frame A1′ and which isdisplayed at a time point of 2/120 [sec], the object moves by MV1 (=α)[pix/(5/120) sec]×2/120 [sec]. In the third frame A1′ which is insertedimmediately after the second frame A1′ and which is displayed at a timepoint of 3/120 [sec], the object moves by MV1 (=α) [pix/(5/120)sec]×3/120 [sec]. In the fourth frame A1′ which is inserted immediatelyafter the third frame A1′ and which is displayed at a time point of4/120 [sec], the object moves by MV1 (=α) [pix/(5/120) sec]×4/120 [sec].

Then, the frame B1 is displayed at a time point of 5/120 [sec]. Theposition of the object in the frame B1 is moved to the right from thereference position of the object in the frame A1 by α [pix].Specifically, the object moves to the right by MV1 (=α)[pix/(5/120)sec]×1/120 [sec] from the position of the object in thefourth frame A1′ which is displayed in the time point of 4/120 [sec].This means that an amount of movement of the object from the time pointof 4/120 [sec] to a time point of 5/120 [sec] is equal to an amount ofprevious movement.

Thereafter, the inserted first to fourth frames B1′, the frame A2, andthe inserted first to fourth frames A2′ are successively displayed witha time interval of 1/120 [sec]. Similarly, in these frames, the objectmoves to the right at constant speed. As a result, a highly-visiblejudder is not generated.

However, the realistic sensation unique to movies is generated due to ajudder generated to some extent. Therefore, there arises the problemthat when the user views such a video image which does not include thejudder in an environment as dark as general movie theaters, therealistic sensation unique to movies is deteriorated.

In other words, since the human eyes have the eye-followingcharacteristic, when judders of identical degrees are generated in avideo image in an environment as dark as general movie theaters and in avideo image in an environment brighter than the general movie theaters,the video image shown in the environment as dark as general movietheaters is recognized as a proper video image having realisticsensation similar to that in the movie theaters whereas the video imageshown in the environment brighter than the general movie theater isrecognized as a video image including a highly-visible judder.

Therefore, in order to reduce a highly-visible judder withoutdeteriorating the realistic sensation similar to that in general movietheaters, that is, without deteriorating excellent characteristics ofthe movie source, the degree of generation of the judder should beappropriately changed in accordance with brightness in the userenvironment. This change is realized by changing the motion vector MV inaccordance with the brightness, which will be described hereinafter. Anamount of the change of the motion vector MV in accordance with thebrightness corresponds to the gain G calculated using the gaincalculation unit 14 in FIG. 1.

The gain calculation unit 14 stores a function f(α) shown in FIG. 6, forexample. The function f(α) is an example obtained when it is assumedthat the eye-following characteristic is based on the proportionalrelationship between the eye sensibility and the brightness. In thisexample, the gain calculation unit 14 assigns the brightness P1 detectedusing the brightness detector 15 to the function f(α) as an input valueα serving as a parameter of the eye sensibility, and supplies an outputvalue f(α) thereof to the motion vector conversion unit 12 as the gainG.

The gain G obtained as described above is used to appropriately generatea judder suitable for the eye-following characteristic of the user(provided that the eye sensibility is in proportion to the brightness)under brightness of the user environment at the time. That is, thefunction f(α) is not limited to the example of FIG. 6, and a function ofa curve in accordance with an actual eye-following characteristic ispreferably employed.

The gain G is supplied to the motion vector conversion unit 12. Then,the motion vector conversion unit 12 converts the motion vector MV intothe motion vector G×MV, and supplies the motion vector G×MV to the framecompensation unit 13.

Accordingly, the frame compensation unit 13 performs the framecompensation processing as shown in FIG. 7 so as to generate 120-Hzimage data which generates a judder suitable for the brightness at thetime as the output signal T1.

Specifically, FIG. 7 shows an example in a case where the brightness P1detected using the brightness detector 15 is determined to be brightnessP1 t shown in FIG. 6. When the brightness P1 corresponds to thebrightness P1 t, a gain G of 0.5 is obtained using the gain calculationunit 14 and is supplied to the motion vector conversion unit 12. Then,the motion vector conversion unit 12 converts the motion vector MV1 intoa motion vector MV1′ (=0.5×MV1), the motion vector MV2 into a motionvector MV2′ (=0.5×MV2), and the motion vector MV3 into a motion vectorMV3′ (=0.5×MV3), and successively supplies the motion vectors MV1′,MV2′, and MV3′ to the frame compensation unit 13. The frame compensationunit 13 performs the following frame compensation processing.

As shown in FIG. 7, the frame compensation unit 13 inserts first tofourth frames A1′, first to fourth frames B1′, and first to fourthframes A2′ after the frames A1, B1, and A2, respectively, so that theobject (represented by rectangular regions shown in FIG. 7) moves inaccordance with the converted motion vectors MV1′, MV2′, and MV3′,whereby a 120-Hz output signal T1 is generated.

Specifically, when a time point in which the frame A1 is displayed isset to “0” and a position α [pix] of the object is set as a referenceposition, a video image corresponding to the output signal T1 isdisplayed such that the object moves as follows.

In the first frame A1′ which is inserted immediately after the frame A1and which is displayed at a time point of 1/120 [sec], the object movesby MV1′ (=0.5×MV1=0.5×α) [pix/(5/120) sec]×1/120 [sec]. In the secondframe A1′ which is inserted immediately after the first frame A1′ andwhich is displayed at a time point of 2/120 [sec], the object moves byMV1′ (=0.5×MV1=0.5×α) [pix/(5/120) sec]×2/120 [sec]. In the third frameA1′ which is inserted immediately after the second frame A1′ and whichis displayed at a time point of 3/120 [sec], the object moves by MV1′(=0.5×MV1=0.5×α) [pix/(5/120) sec]×3/120 [sec]. In the fourth frame A1′which is inserted immediately after the third frame A1′ and which isdisplayed at a time point of 4/120 [sec], the object moves by MV1′(=0.5×MV1=0.5×α) [pix/(5/120) sec]×4/120 [sec].

Then, the frame B1 is displayed at a time point of 5/120 [sec]. Theposition of the object in the frame B1 is moved from the referenceposition of the object in the frame A1 by α[pix]. That is, the objectmoves by α−MV1′ (=0.5×MV1=0.5×α) [pix/(5/120)sec]×4/120 [sec]=(3/5)×α[pix] from the position of the object in the fourth frame A1′ which isdisplayed in the time point of 4/120 [sec].

The object moves by (1/10)×α [pix] every 1/120 [sec]in a range from thetime point of 1/120 [sec] to the time point of 4/120 [sec]. That is, theobject moves at constant speed. However, from the time point of 4/120[sec] to the time point of 5/120 [sec], the object moves by (3/5)×α[pix]. Accordingly, an amount of movement increases and a judder isgenerated. Note that the amount of the movement is reduced to half theamount of the movement shown in FIG. 4 (that is, similar to the gain Greduced to half thereof), and therefore, a degree of generation of thejudder is reduced. Specifically, in the brightness P1 t detected usingthe brightness detector 15, a judder suitable for the eye-followingcharacteristic of the user is generated. In other words, a judder enoughto realize the realistic sensation unique to movies and which is nothighly visible is generated.

Thereafter, the object is basically displayed similarly to the movementof the object described above. That is, the object moves by (1/10)×α[pix] every 1/120 [sec] in a range from the time point of 5/120 [sec] toa time point of 9/120 [sec] at constant speed. Then, the object moves by(3/5)×α [pix] from the time point of 9/120 [sec] to the time point of10/120 [sec]. Furthermore, the object moves by (1/10)×α [pix] every1/120 [sec] in a range from the time point of 10/120 [sec] to a timepoint of 1/120 [sec]. Then, the object moves by (3/5)×α [pix] from thetime point of 14/120 [sec] to a time point of 15/120 [sec]. Thismovement is repeatedly performed thereafter.

Although not shown, when the brightness P1 detected using the brightnessdetector 15 is changed, the gain G is changed in accordance with thechange of the brightness P1. Therefore, an amount of movement of theobject and a degree of generation of a judder are changed by an amountof change of the gain G.

In other words, when the brightness P1 detected using the brightnessdetector 15 becomes equal to or smaller than predetermined brightness,since a room turns dark, for example, the gain G becomes zero as shownin FIG. 6. Therefore, the motion vector G×MV obtained through conversionperformed using the motion vector conversion unit 12 becomes zero. Inthis case, the output signal T1 generated through the frame compensationprocessing performed using the frame compensation unit 13 corresponds tothe signal shown in FIG. 4.

On the other hand, when the brightness P1 detected using the brightnessdetector 15 becomes equal to or larger than predetermined brightness,since a room is lit up, for example, the gain G of 1 is obtained asshown in FIG. 6. Therefore, the motion vector G×MV obtained through theconversion performed using the motion vector conversion unit 12corresponds to the motion vector MV which has not yet been subjected tothe conversion. In this case, the output signal T1 generated through theframe compensation processing performed using the frame compensationunit 13 becomes equal to the signal shown in FIG. 5.

As described above, a judder suitable for the brightness P1 detectedusing the brightness detector 15 is generated. That is, in thebrightness P1, a judder enough to realize the realistic sensation uniqueto movies and which is not highly visible is generated.

Note that a degree of the judder enough to realize the realisticsensation unique to movies and which is not highly visible issubjectively determined by the user. Therefore, the user correction unit16 is provided in the example shown in FIG. 1 so that desired juddersare generated for individual users. Specifically, the user freelychanges the correction amount ΔG supplied from the user correction unit16 to the gain calculation unit 14, Accordingly, the gain G is changedin accordance with the correction amount ΔG desired by the user.Consequently, a degree of generation of a judder is freely controlled bythe user.

The operation of the image processing apparatus shown in FIG. 1 in thecase where the film is supplied as the input signal T0 has beendescribed above.

The operation of the image processing apparatus shown in FIG. 1 in thecase where the film which has been subjected to the 2-3 pull downprocessing is supplied as the input signal T0 will now be describedhereinafter with reference to FIGS. 8 to 11.

FIG. 8 is a timing chart illustrating a movement of an object in a filmwhich has been subjected to the 2-3 pull down processing represented bythe input signal T0.

FIG. 9 is a timing chart illustrating a movement of the object relativeto the output signal T1 obtained after the frame compensation unit 13performs frame compensation processing in a case where all the motionvectors MV1′, MV2′, and MV3′ obtained after conversion processing become0, that is, in a case where the gain G becomes 0 when a room turns dark,for example.

FIG. 10 is a timing chart illustrating a movement of the object relativeto the output signal T1 obtained after the frame compensation unit 13performs the frame compensation processing in a case where all themotion vectors MV1′, MV2′, and MV3′ obtained after the conversionprocessing become half the motion vectors which have not yet beensubjected to the conversion processing, that is, in a case where thebrightness P1 t is detected using the brightness detector 15 and thegain G becomes 0.5.

FIG. 11 is a timing chart illustrating a movement of the object relativeto the output signal T1 obtained after frame compensation unit 13performs the frame compensation processing in a case where all themotion vectors MV1′, MV2′, and MV3′ obtained after the conversionprocessing become equal to the motion vectors which have not yet beensubjected to the conversion processing, that is, the gain G becomes 1when a room is lit up, for example.

When FIGS. 8 to 11 are compared with FIGS. 3, 4, 7, and 5, the followingfact becomes apparent. The operation of the image processing apparatusshown in FIG. 1 in the case where the film which has been subjected tothe 2-3 pull down processing is supplied as the input signal T0 is“basically” the same as the operation of the image processing apparatusin the case where the film is supplied as the input signal T0 which isdescribed with reference to FIGS. 3 to 5 and FIG. 7. They are“basically” the same except for the fact that since a frame period ofthe input signal T0 which has not yet been subjected to the frameconversion is slightly different between the two cases, the number offrames to be inserted is slightly different between the two cases, forexample.

As described above, the image processing apparatus shown in FIG. 1appropriately controls an amount of generation of a judder in accordancewith brightness of a user environment (a room, for example) where theuser actually views a movie source, that is, in accordance with aneye-following characteristic of human eyes by performing the operationdescribed above. Consequently, excellent characteristics of the moviesource are brought out.

Note that a series of the processes described above (or a part of theprocesses) may be executed by hardware or software.

In this case, the entire image processing apparatus of FIG. 1 or a partof the image processing apparatus of FIG. 1 may be constituted by acomputer shown in FIG. 12, for example.

In FIG. 12, a CPU (Central Processing Unit) 101 performs variousprocesses in accordance with a program recorded in a ROM (Read OnlyMemory) 102 or a program loaded from a storage unit 108 into a RAM(Random Access Memory) 103. The RAM 103 also stores therein data whichis used when the CPU 101 performs the various processes.

The CPU 101, the ROM 102, and the RAM 103 are connected with one anotherthrough a bus 104. The bus 104 is further connected to an input/outputinterface 105.

The input/output interface 105 is connected to an input unit 106including a keyboard and a mouse, an output unit 107 including adisplay, the storage unit 108 including a hard disk, and a communicationunit 109 including a modem and a terminal adapter. The communicationunit 109 communicates with other apparatuses through a network includingthe Internet.

The input/output interface 105 is further connected to a drive 110. Aremovable medium 111 including a magnetic disk, an optical disc, amagneto-optical disc, and a semiconductor memory is appropriatelyinserted into the drive 110. A computer program read from the removablemedium 111 is appropriately installed in the storage unit 108.

When the series of the processes are performed using software, a programincluded in the software is installed into a computer incorporated in adedicated hardware or into a general personal computer capable ofperforming various functions by installing various programs, forexample, through a network or a recording medium.

Such a recording medium including the program is constituted by theremovable medium (package medium) 111 such as a magnetic disk (includinga floppy disk), an optical disc (including a CD-ROM (Compact Disk-ReadOnly Memory), a DVD (Digital Versatile Disk)), a magneto-optical disc(including an MD (Mini-Disk)) or a semiconductor memory, in which theprogram is recorded, which is provided separately from the apparatusbody, and which is distributed so as to supply the program to the user.Alternatively, the recording medium including the program is constitutedby the ROM 102 including the program recorded therein to be supplied tothe user in a state in which the ROM is incorporated in the apparatusbody, or a hard disk included in the storage unit 108.

Note that in this specification, steps of describing the program in therecording medium may be performed in a time series manner in an order ofthe steps, or may be performed in parallel or individually.

As described above, in this specification, a system means the entireapparatus including a plurality of processing devices and processingunits.

In the foregoing examples, the movement amount MV indicates an amount ofmovement in a certain direction, that is, in a horizontal direction inthe frame, for example, for simplicity. However, it is apparent that themovement amount MV may indicate an amount of movement in otherdirections. In this case, that is, in a case where the movement amountMV indicates an amount of movement in a direction other than thehorizontal direction, since the image processing apparatus sets themovement amount as a vector, processes basically similar to the variousprocesses described above are executed.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

1. An image processing apparatus which performs processing on an inputmoving image including a plurality of access units arranged every firstperiod, the image processing apparatus comprising: motion vectorcalculation means for calculating a motion vector of an object includedin the input moving image every second period; motion vector conversionmeans for converting the motion vector by multiplying the motion vectorcalculated using the motion vector calculation means by a predeterminedgain; frame compensation means for generating an output moving imageincluding a plurality of access units arranged every third period byperforming frame compensation processing of converting or compensatingfor the access units on the input moving image so that the object movesin accordance with the motion vector which has been subjected to theconversion performed using the motion vector conversion means; and gaincalculation means for calculating the predetermined gain in accordancewith brightness in a user environment where a user views the outputmoving image and supplying the obtained gain to the motion vectorconversion means.
 2. The image processing apparatus according to claim1, further comprising: brightness detection means for detecting thebrightness in the user environment, wherein the gain calculation meanscalculates the gain in accordance with the brightness detected using thebrightness detection means.
 3. The image processing apparatus accordingto claim 1, further comprising: correction means for correcting the gaincalculated using the gain calculation means in accordance with aninstruction issued by the user.
 4. An image processing method,implemented by a processor, of an image processing apparatus whichperforms processing on an input moving image including a plurality ofaccess units arranged every first period, the image processing methodcomprising the steps of: calculating, by the processor, a motion vectorof an object included in the input moving image every second period;converting the motion vector by multiplying the calculated motion vectorby a predetermined gain; generating an output moving image including aplurality of access units arranged every third period by performingframe compensation processing of converting or compensating for theaccess units on the input moving image so that the object moves inaccordance with the motion vector which has been subjected to theconversion; and calculating, by the processor, the predetermined gain inaccordance with brightness in a user environment where a user views theoutput moving image.
 5. A non-transitory computer readable mediumencoded with a program which causes a computer that performs processingon an input moving image including a plurality of access units arrangedevery first period to execute, the program comprising the steps of:calculating a motion vector of an object included in the input movingimage every second period; converting the motion vector by multiplyingthe calculated motion vector by a predetermined gain; generating anoutput moving image including a plurality of access units arranged everythird period by performing frame compensation processing of convertingor compensating for the access units on the input moving image so thatthe object moves in accordance with the motion vector which has beensubjected to the conversion; and calculating the predetermined gain inaccordance with brightness in a user environment where a user views theoutput moving image.
 6. An image processing apparatus which performsprocessing on an input moving image including a plurality of accessunits arranged every first period, the image processing apparatuscomprising: a processor and a memory; a motion vector calculation unitconfigured to calculate a motion vector of an object included in theinput moving image every second period; a motion vector conversion unitconfigured to convert the motion vector by multiplying the motion vectorcalculated using the motion vector calculation unit by a predeterminedgain; a frame compensation unit configured to generate an output movingimage including a plurality of access units arranged every third periodby performing frame compensation processing of converting orcompensating for the access units on the input moving image so that theobject moves in accordance with the motion vector which has beensubjected to the conversion performed using the motion vector conversionunit; and a gain calculation unit configured to calculate thepredetermined gain in accordance with brightness in a user environmentwhere a user views the output moving image and supply the obtained gainto the motion vector conversion unit.